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Subtle  flaws  in  design  can  have  multimillion-dollar  effects— and  sometimes 
potentially  catastrophic  ones.  A  vigorous  logistics  test  and  evaluation  process 
is  essential  to  ferret  out  these  potential  problems,  and  make  sure  that  when 
fielded  a  system  is  supportable,  maintainable,  safe,  survivable,  and 
transportable.  An  Office  of  the  Secretary  of  Defense  (OSD)  study  suggests 
that  this  is  a  job  we  need  to  do  better;  here  is  a  guide  on  how  to  go  about  it, 
illustrated  with  specific  cases  that  show  its  value. 


Logistics  is  a  primary  element  in  a  thor¬ 
ough  test  and  evaluation  of  new  or 
modified  weapons  systems.  It  is  criti¬ 
cal  to  the  warfighter  that  we  provide  an 
early  assessment  of  the  overall  support  char¬ 
acteristics  of  the  system  we  are  evaluating. 

In  our  evaluations  of  the  system  under 
test  (SUT),  we  need  to  remain  focused  on 
the  goal  of  providing  a  system  that  maxi¬ 
mizes  its  operational  availability  (A  ) 
within  the  life-cycle  cost  (LCC)  of  the 
program.  The  system  must  minimize  the 
amount  of  time  that  maintainers  expend 
getting  it  ready  to  perform  its  mission, 
recovering  the  system  after  the  mission, 
and  returning  it  to  a  mission-ready  status 
again.  That  is,  we  must  build  systems  that 
maximize  their  Aos  or  we  build  a  ground 
target  for  the  enemy. 


A  key  principle  of  logistics  test  and 
evaluation  (LT&E)  is  to  ensure  that  the 
system  under  development  is  able  to 
achieve  the  readiness  objectives  for  both 
peacetime  and  wartime  scenarios.  This  is 
accomplished  through  a  unified  and  itera¬ 
tive  approach  to  the  management  and  tech¬ 
nical  activities  that  allow  support  consid¬ 
erations  to  influence  system  requirements 
and  the  design  process.  The  support 
requirements  and  the  design  must  be 
optimally  related  to  each  other  in  order  to 
maximize  the  system  Ao  throughout  its  life 
cycle.  During  the  LT&E  process,  we  sys¬ 
tematically  evaluate  each  of  the  logistic 
support  elements  and  their  interrelation¬ 
ship  with  other  elements.  Part  of  this  pro¬ 
cess  is  to  identify  system  deficiencies, 
provide  enhancement  options,  and  then 
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evaluate  corrections  made  to  the  system. 
This  continuous  process  will  help  to  en¬ 
sure  the  system  maximizes  its  Ao  when  it 
is  fielded. 

So  what  drove  us  to  reexamine  the 
method  we  used  to  evaluate  the  SUT?  One 
impetus  was  the  OSD  study  that  suggested 
the  Services  were  not  systematically 
evaluating  the  SUT  as  well  as  could  be 
done.  While  we  may  have  thought  we  were 
doing  a  good  job  of  putting  systems  out 
in  the  field  that  were  reliable  and  main¬ 
tainable  and  able  to  achieve  their  A  ,  the 
feedback  from  the  field  was  that  we  could 
do  much  better. 

Reliability  and  maintainability  are  what 
really  determine  Ao.  Recall  the  definition 

of  A  : 

0 

MTBM 

A0=  - 

MTBM  +  MDT 

where  MTBM  equals  the  mean  time  be¬ 
tween  maintenance,  and  MDT  equals  the 
maintenance  delay  time.  The  formula  is 
based  on  the  assumption  that  the  system 

is  designed  with 

"Reliability  and  reliability.  Reli- 

rraintainability  ability  is  the 

along  with  perfor-  probability  that 

manae,  should  ad  an  item  will 

as  equal  partners  perform  a  re¬ 
in  the  require-  quired  function 

ments  generation  underspecified 

P1-0005®1”  conditions  for  a 

specified  pe¬ 
riod  of  time,  or  at  a  given  point  in  time. 
Maintainability  is  the  probability  that  an 
item  will  be  retained  in,  or  restored  to,  a 
specified  condition  within  a  given  period 
of  time.  Reliability  and  maintainability, 
along  with  performance,  should  act  as 


equal  partners  in  the  requirements  genera¬ 
tion  process. 

Life-cycle  Costs 


In  a  typical  acquisition  program,  the 
system  program  office  (SPO)  has  commit¬ 
ted  more  than  80  percent  of  the  LCC 
before  the  engineering  and  manufacturing 
development  test  program  has  begun.  The 
program  hasn’t  written  the  check,  but  they 
have  made  key  decisions  on  the  design  of 
the  system  and  the  support  characteristics 
that  essentially  commit  those  funds.  Based 
on  design  and  its  reliability  and  maintain¬ 
ability  predictions,  the  SPO  will  determine 
of  the  number  of  spares  of  each  particular 
type  that  will  be  purchased,  what  support 
equipment  will  be  used,  and  whether  new 
equipment  will  be  procured.  These  pre¬ 
dictions  are  also  used  to  determine  the 
types  of  skills  needed  and  the  varying  skill 
levels  required,  and  other  manpower  con¬ 
siderations.  During  LT&E,  we  are  char¬ 
tered  to  validate  the  contractor’s  estimates 
and  provide  that  data  to  the  program  office. 
Under-  or  over-estimating  the  reliability 
and  maintainability  of  the  component 
pieces  of  the  system  will  cause  already 
limited  dollars  to  be  allocated  unwisely. 
Accomplishing  LT&E  early  and  continu¬ 
ously  throughout  the  development,  test, 
and  evaluation  (DT&E)  program  allows 
SPO  personnel  to  determine  what  needs 
to  be  adjusted  and  by  how  much,  to 
provide  the  required  system  support 
throughout  the  program’s  life  cycle. 

Most  of  the  LCC  of  a  system  is  spent 
during  the  operations  and  support  (O&S) 
phase.  Typically  10  percent  of  the  pro¬ 
gram’s  LCC  is  spent  during  the  research, 
development,  test,  and  evaluation  (RDT&E) 
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phase  of  the  program.  While  we  may  view 
production  as  the  most  costly  portion  of 
the  program  per  unit  of  time,  it  really  only 
amounts  to  some  30  percent  of  the  LCC. 
Based  on  these  figures,  it  becomes  readily 
apparent  that  the  largest  cost  driver  in  the 
life  of  a  system  is  the  O&S  phase.  With  a 
system  such  as  the  B-52  bomber,  which 
is  doubling  (or  more)  its  life  expectancy, 
it  is  easy  to  see  that  O&S  costs  will  form 
75-90  percent  of  its  LCC.  Essentially,  it 
costs  almost  twice  as  much  to  maintain 
and  support  a  system  once  it  has  been 
fielded  as  it  does  to  acquire  and  produce 
that  same  system.  For  a  major  system  such 
as  the  B-2,  which  costs  $44  billion  in 
RDT&E  and  production,  the  Air  Force  can 
expect  to  spend  about  $66  billion  more  in 
the  O&S  portion  of  that  aircraft’s  life 
cycle. 


Coordinating  the  LT&E  Process 

The  LT&E  of  today  is  a  significantly 
different  process.  Previously  we  used 
maintainability  demonstrations,  support 
equipment  compatibility  demonstrations, 
and  technical  order  verifications,  but  all 
were  conducted  independently.  The  cur¬ 
rent  LT&E  discipline  makes  a  synergistic 
application  of  all  the  logistics  support 
elements.  The  goal  of  LT&E  is  to  test, 
evaluate,  analyze  the  system,  and  then 
report  those  findings  to  the  customer. 
Established  guidance  exists  that  addresses 
the  need  to  accomplish  LT&E.  (In  fact, 
when  Air  Force  Instruction  [AF1]  99-101 
was  being  drafted,  we  were  asked  to 
develop  the  wording  for  LT&E.) 

Now  that  we  have  guidance,  what  do 
we  use  for  test  and  evaluation  criteria? 
What  will  define  satisfactory  systems 


The  B-52  Borrber 
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performance?  The  place  to  start  is  the 
operational  requirements  document 
(ORD).  What  are  the  requirements  the  user 
has  given  us  for  the  system?  How  have 
those  requirements  been  translated  into 
the  system  specification?  For  example, 
what  are  the  mean  time  between  failure 
(MTBF)  and  mean  time  to  repair  (MTTR) 
requirements? 

Human  factor  requirements  come  from 
both  the  ORD  and  the  system  specifica¬ 
tion.  For  example,  will  we  be  able  to 
complete  all  maintenance  tasks  while  in 
chem  gear,  or  even  cold  weather  gear?  Is 
equipment  being  designed  for  ease  of  ac¬ 
cessibility  and  maintenance?  Are  line-re¬ 
placeable  units  (LRUs)  being  designed  and 
installed  in  such  a  way  that  one  person 
can  accomplish  the  task  if  required? 

Supportability  analysis  predictions  are 
a  critical  part  of  how  we  evaluate  the  sys¬ 
tem.  If  the  LRU  has  been  designed  to  work 
for  350  hours  before  it  fails,  then  that  is 
the  criteria  used  to  evaluate  its  success  or 
failure.  If  it  fails  continually  at  100  hours, 


the  customer  needs  to  know,  in  order  to 
reevaluate  system  design.  If  spare  LRUs 
were  purchased  based  on  a  350-hour 
MTBF,  there  would  now  be  an  insufficient 
quantity  of  spares. 

We  need  to  ensure  that,  although  we 
test  each  part  of  the  system,  we  are  able 
to  evaluate  the  overall  system  supportabil¬ 
ity,  maintainability,  safety,  survivability, 
and  transportability.  You  can  almost 
directly  translate  these  requirements  into 
the  logistic  support  elements  (LSEs). 

When  comparing  the  test  data  with  the 
LSE  predictions,  be  aware  that  a  new 
design  or  new  use  of  an  LRU  may  suffer 
infant  mortality  or  have  initially  low  reli¬ 
ability.  Test  the  system  long  enough  to  dis¬ 
tinguish  between  development  pains  and 
fundamental  problems  (i.e.,  the  unit  not 
working  as  it  should  and  the  data  telling 
you  it’s  not  getting  any  better).  Report 
these  problems  to  the  customer  and  also 
provide  them  with  recommended  changes 
to  improve  the  shop  replaceable  unit 
(SRU)/LRU,  or  system.  Technical  Order 
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00-35D-54  provides  not  only  a  means  to 
report  system,  LRU,  and  SRU  deficien¬ 
cies,  but  a  means  to  recommend  enhance¬ 
ments  to  the  system.  This  is  a  reason  why 
we  use  people  who  have  current  opera¬ 
tional  experience  to  perform  the  LT&E. 
They  know  what  is  happening  in  the  field 
and  know  how  to  make  the  system  better. 

Finding  Problems  Early 

The  AFI 99-101  stressed  that  the  DT&E 
communities  look  at  the  maintainability 
and  suitability  (supportability)  of  the  sys¬ 
tem  early  in  the  development  program. 
Although  at  this  point  we  may  be  assess¬ 
ing  prototypes  or  preproduction  equip¬ 
ment,  we  can  still  make  an  early  determi¬ 
nation  of  how  the  system  will  work  when 
updated,  modified,  or  changed,  based  on 
our  DT&E  findings.  Consider  the  train¬ 
ing  that  was  provided  to  the  maintenance 
team  and  make  determinations  by  their 
ability  to  maintain  the  system:  Was  the 
training  adequate?  Are  the  technical  orders 
they  are  working  with  adequate  to  accom¬ 
plish  the  tasks?  (They  may  not  be  validated 
and  verified.)  Is  the  support  equipment 
they  are  using  the  equipment  that  will  be 
fielded?  If  not,  have  they  been  able  to 
evaluate  the  proposed  support  equipment 
designs?  What  is  the  number  of  person¬ 
nel  needed  to  support  and  test  the  system 
in  the  test  environment;  will  this  number 
be  adequate  for  the  system  when  it  is 
fielded?  Do  we  have  the  correct  Air  Force 
Speciality  Codes  (AFSCs)  identified  with 
the  correct  skill  levels?  Finally,  are  the 
maintenance  procedures  correct?  Are  we 
asking  an  engine  technician  to  maintain 
an  instrument  system,  just  because  it  is 
an  engine  instrument? 


In  a  typical  program  it  is  many  years 
from  the  development  of  the  ORD  to  the 
point  at  which  a  system  is  available  for 
testing  in  the  engineering  and  manufac¬ 
turing  development  phase.  During  that 
time,  events  can  change  the  user  require¬ 
ments:  The  threat  may  change,  or  the 
program  funding  may  be  reduced  or 
increased.  The  ORD  format  will  identify 
all  of  the  LSEs 
and  will  provide  „LittIe 

the  author  the  ^  ^  eM^iy 

opportunity  to  inpir^e  ipon  money 

input  require-  that  vue  vue  need 

ments  into  the  to  spend  on  otiher 

ORD  fortesting  new  equipment 

those  elements  Wadea" 
to  ensure  that  a 

supportable  system  is  fielded.  The  inser¬ 
tion  of  human  factors  and  LS  A  predictions 
is  another  way  to  evaluate  the  data  that 
we  gather  on  the  system.  These  items,  the 
10  LSEs,  the  user’s  requirements,  human 
factors,  and  the  supportability  analysis 
predictions,  when  tested  and  evaluated 
together,  provide  a  comprehensive  analy¬ 
sis  of  whether  the  system,  when  fielded, 
will  be  able  to  attain  the  required  Ao. 

Other  documents  provide  additional 
information.  The  program  office  will 
develop  the  logistics  support  plan,  which 
describes  how  the  customer  and  the 
acquisition  personnel  envision  life-cycle 
support  for  the  system.  LT&E  is  the  pri¬ 
mary  method  used  to  test  the  plan  and 
determine  if  it  is  effective.  The  contractor 
will  also  have  a  plan  describing  how  the 
system  will  be  supported  from  production 
to  disposal.  This  plan  should  be  developed 
using  a  relational  data  base,  which  flags 
how  changes  in  one  element  will  affect 
others.  (For  example,  if  the  numbers  and 
skills  of  the  personnel  in  the  field  change, 
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the  data  base  will  identify  other  support 
elements  that  will  be  affected.) 

The  test  and  evaluation  master  plan 
(TEMP),  written  by  the  test  planning  work¬ 
ing  group,  is  one  of  the  primary  manage¬ 
ment  documents  for  this  phase  of  the 
program.  The  TEMP  contains  five  sections: 

•  a  system  description; 

•  an  integrated  program  schedule  with  a 
focus  on'  the  test  schedule; 

•  DT&E  objectives; 

•  the  objectives  of  the  operational  testers ; 
and 

•  the  test  program  resource  requirements . 

Examples  of  program  resources  are: 
number  of  personnel  required,  location  of 
testing,  duration  of  testing,  number  of  test 
articles,  number  of  spare  assets,  funding 
by  fiscal  year,  and  training.  Program 
resources  are  what  we,  the  test  community, 
need  to  complete  the  test  and  evaluation 
of  the  system. 

LT&E  Successes 


Let’s  look  for  a  moment  at  some  of  the 
results  of  accomplishing  LT&E.  During 
testing  of  a  new  F-15  crypto  key  loading 
device,  the  contractor  had  recommended 
a  new  extension  cable.  Maintenance  per¬ 
sonnel  determined  that  it  wasn’t  necessary 
to  use  the  cable  to  load  the  device.  This 
saved  $58  per  cable  and  about  $1,044.00 
per  F-15  fighter  wing  per  year.  It  doesn’t 
seem  like  much,  until  you  look  at  a  small 
cost  like  this  in  the  context  of  shrinking 


budgets.  How  many  fighter  wings  of  F- 
15s  do  we  have,  and  how  long  will  the  F- 
15  remain  in  the  inventory?  Little  costs 
mount  up,  and  eventually  impinge  upon 
money  that  we  we  need  to  spend  on  other 
new  equipment  or  upgrades. 

The  results  of  successful  LT&E  can  be 
potentially  much  more  dramatic:  In  the 
original  global  positioning  satellite  modi¬ 
fications  on  the  F-15,  the  seal  for  the  con¬ 
trolled  reception  pattern  antenna  had  metal 
washers  that  were  not  captive  in  the  seal 
and  were  prone  to  slip  off  the  seal  during 
installation  and  removal.  As  a  result  of 
input  from  the  maintainers,  the  finished 
product  eliminated  the  need  for  the  seal 
and  the  loose  hardware.  Another  modifi¬ 
cation  of  the  global  positioning  satellite 
recommended  by  maintenance  personnel 
was  the  relocation  of  the  proposed  antenna 
electronics  unit,  which  chafed  against  the 
right  rudder  pedal  cable.  The  possible  con¬ 
sequences  of  this  initial  design  flaw — loss 
of  the  aircraft  and  possibly  of  the  crew — 
demonstrate  the  value  of  this  kind  of 
assessment. 

An  example  of  what  can  happen  if  the 
maintainers  are  not  involved  early  in  the 
DT&E  process  is  demonstrated  by  the 
modified  SUU-73/A  pylon.  When  the 
pylon  was  installed  as  designed,  certain 
engine  access  panels  could  not  be  opened. 
So  every  time  access  was  required,  the  fuel 
tank  pylon  or  the  weapons  pylon  would 
have  to  be  removed — an  8-manhour  job. 
Needing  excess  time  to  repair  aircraft  is 
just  the  same  as  not  having  enough  aircraft 
to  accomplish  the  mission. 

Here’s  a  good  example  of  how  being 
involved  early  in  the  acquisition  process 
has  an  impact  on  the  design.  An  electro- 
environmental  specialist  looking  at  the 
mockup  at  the  plant  in  Long  Beach,  CA, 
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The  F-15 


noticed  that  the  C-17  cargo  compartment 
liquid  oxygen  filler  connection  had  pro¬ 
truding  screws,  which  would  prevent  the 
filler  cart  hose  from  being  attached  prop¬ 
erly.  When  brought  to  the  attention  of  the 
manufacturer,  the  flaw  was  described  as 
merely  an  error  in  the  mockup.  Through 
persistence,  the  specialist  found  and  stud¬ 
ied  the  design  drawing,  and  determined 
that  it  was  assembled  correctly.  The  con¬ 
nector  would  not  have  been  able  to  lock, 
and  maintenance  would  have  been  unable 
to  service  the  system.  The  drawing  was 
changed  and  the  assembly  line  was 


checked  to  ensure  that  the  change  was 
incorporated. 

The  following  example  shows  how  a 
small  item  can  run  into  big  money  because 
reliability  figures  weren’t  accurately  esti¬ 
mated.  A  wingtip  lamp  for  the  C-17,  both 
front  and  rear  positions,  is  the  same  lamp 
that  is  installed  in  the  KC-10  (DC-10)  to 
illuminate  the  wing’s  leading  edge  during 
flight.  On  the  KC-10  it  is  mounted  in 
the  side  of  the  fuselage  skin  in  a  very 
benign  environment,  and  needs  to  be 
replaced  every  676  hours.  That  same  lamp 
on  the  wingtip  of  the  C-17  is  in  a  different 
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operating  environment  (higher  vibration 
and  temperatures)  and  the  lamp  only  lasts 
for  25  hours  in  the  forward  position  and 
50  hours  in  the  aft  position. 

The  lamp  did  not  come  close  to  its  pre¬ 
dicted  value,  and  if  insufficient  spares  of 
this  mission-critical  part  were  purchased, 
all  of  the  C-17s  could  soon  be  grounded 
for  want  of  a  lamp.  The  impact  of  the 
lamp’s  invalid  reliability  figure  of  676 
hours  MTBF  on  the  life-cycle  cost  of  the 
C-17  is  an  estimated  additional  $4.5  mil¬ 
lion.  With  the  actual  MTBF  numbers  in 
hand,  the  system  program  office  can  at 
least  adjust  the  procurement  requirements 
for  spares. 

Here  is  another  example  of  the  effec¬ 
tiveness  of  early  LT&E  in  identifying 
design  shortfalls.  The  original  estimate  for 
the  C-17  water  coalescer  bag  was  that  it 
would  be  changed  approximately  once  a 
year  (about  every  1,000  to  1,200  flying 


hours),  which  was  the  normal  change  rate 
for  the  current  fleet  of  airlifters  (C-130, 
C-141,  C-5).  We  determined  during  the 
test  program  that  because  of  the  design  of 
the  auxiliary  power  unit’s  intake  and 
exhaust,  and  the  proximity  to  the  air  con¬ 
ditioning  system  intake,  the  bags  (some¬ 
times  called  the  water  separator  socks) 
were  becoming  clogged  every  25  flying 
hours.  It  didn’t  take  long  for  supply  of  bags 
to  run  out.  Through  some  redesign  and 
changes  in  the  maintenance  procedures, 
the  bags  now  last  about  250  hours  (still  a 
dramatic  LCC  impact).  If  it  doesn’t  get 
fixed,  the  cost  will  be  an  estimated  $29 
million. 

Another  instance  of  the  value  of  early 
design  review  occurred  during  the  B-2  pro¬ 
gram,  when  egress  technicians  at  the 
manufacturing  facility  reviewing  the 
mockup  design  noticed  the  upper  escape 
hatch  was  secured  with  hi-torq  fasteners. 


280 


Logistics  Test  and  Evaluation:  An  Overview 


The  upper  hatch  has  to  be  removed  once  a 
year  for  inspection  of  the  ejection  seat.  The 
maintenance  demonstrations  showed  that 
with  the  hi-torqs  installed,  it  would  take 
two  people  214  manhours  to  remove  the 
hatch.  This  amounted  to  a  5-day  effort, 
working  around  the  clock.  The  egress  tech¬ 
nicians  wanted  to  change  the  hi-torqs  for 
screws  and  nutplates,  but  the  engineers 
were  reluctant,  asserting  that  the  hatch  was 
a  stress  panel.  The  egress  technicians 
pointed  out  that  the  F-4  aircraft  was  basi¬ 
cally  a  flying  stress  panel,  and  used  screws 
and  nutplates  predominately  throughout 
the  aircraft  to  secure  panels.  It  was  agreed 
to  evaluate  the  change  to  screws  and 
nutplates.  After  a  series  of  successful  tests 
at  the  Holloman  Air  Force  Base,  NM,  sled 
track,  it  was  agreed  to  use  screws  and 
nutplates.  The  maintenance  task  time  was 
reduced  to  18  manhours  (2  people) — in 
other  words,  one  normal  work  shift.  This 
seemingly  simple  change  meant  the  air¬ 
craft  increased  its  Ao  dramatically.  What 
this  means  is  the  inspection  can  be  done 


overnight  and  the  aircraft  will  be  ready  to 
fly  and  fight  again  the  next  day. 

Conclusion 


Logistics  test  and  evaluation  is  a  major 
portion  of  the  DT&E  process.  The  goal  is 
to  field  systems  that  are  supportable  from 
the  very  beginning.  That  doesn’t  mean  that 
we  won’t  continue  to  modify  weapons 
systems.  It  means  that  the  modifications 
we  install  will  be  the  result  of  new  tech¬ 
nology,  not  something  that  we  will  con¬ 
stantly  have  to  modify  just  to  meet  the 
original  reliability  and  maintainability 
goals.  We  will  make  sure  the  user  gets 
what  they  ask  for  and  that  it  meets  the 
ORD  requirements  for  Ao.  If  it  is  supposed 
to  have  a  95  percent  fully  mission  capable 
rate,  is  that  what  is  occurring  in  the  field? 
If  it  is  not  at  95  percent,  why  not?  Does 
the  radar  work  for  500  hours  before  it  fails, 
as  it  was  required  to  do?  If  not,  why  not? 
Is  the  user  spending  all  of  its  O&S  money 
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on  modifying  the  system  to  make  it  meet 
those  original  requirements,  or  are  they 
spending  it  on  the  things  necessary  to 
maintain  their  proficiency  to  be  prepared 
in  times  of  crisis? 

Ensuring  that  the  warfighter  receives 
maintainable  and  supportable  systems  is 


the  goal  of  LT&E.  While  it  will  generally 
have  a  larger  LCC  impact  if  discovered 
during  the  DT&E  process.  LT&E  should 
continue  throughout  the  operational 
T&E  phase  and  in  fact  should  be  a  part 
of  any  modifications  that  will  effect  the 
supportability  of  the  system. 
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